pipsCloud: high performance cloud computing for remote sensing big data management and processing

摘要

With the increasing requirement of accurate and up-to-date resource & environmental information for regional and global monitoring, large-region covered multi-temporal, multi-spectral massive remote sensing (RS) datasets are exploited for processing. The remote sensing data processing generally follows a complex multi-stage processing chain, which consists of several independent processing steps subject to types of RS applications. In general the RS data processing for regional environmental and disaster monitoring are recognized as typical both compute-intensive and data-intensive applications.To solve the aforementioned issues efficiently, we propose pipsCloud which combine recent Cloud computing and HPC techniques to enable large-scale RS data processing system as on-demand real-time services. Benefiting from the ubiquity, elasticity and high-level of transparency of Cloud computing model, the massive RS data managing and data processing for dynamic environmental monitoring are all encapsulate as Cloud with Web interfaces. Where, a Hilbert-R+ based data indexing mechanism is employed for optimal query and access of RS imageries, RS data products as well as interim data. In the core platform beneath the Cloud services, we provide a parallel file system for massive high-dimensional RS data and offers interfaces for intensive irregular RS data accessing so as to provide improved data locality and optimized I/O performance. Moreover, we adopt an adaptive RS data analysis workflow manage system for on-demand workflow construction and collaborative execution of distributed complex chain of RS data processing, such as forest fire detection, mineral resources and coastline monitoring. Through the experimental analysis we have show the efficiency of the pipsCloud platform.